Pulls pip information and OS installed packages and check if the installed version has a registered CVE.
The tool knows how to find relevant CVE for a program and remove rejected CVEs.
I found this tool very powerful to detect system vulnerable programs versions, although there are some false positives due to the challenges.
Running time: it took me around 5 minutes for 108 python-pip packages. The bootle-neck is the online Circl API (More about the API later on).
The project writtenin python3.
NPM has integrated the audit tool into the program. This tool can show the number of vulnerable packages you are using in your project and help you to update the packages to a newer-secured version (without breaking the dependencies). I thought it will be interesting to know how vulnerable my current python-pip packages. In contrast to npm, the pip's packages are installed globally and therefore more vulnerable when using on a couple of projects (unless you use conda environments). Also, the pip package manager does not encourage the users to update the packages.
Later I extend the tool to support searching CVEs for Windows programs, macOS, and ubuntu programs.
Think about a system that you do not frequently use. For example a Linux partition (or WSL) on your regular PC you do not frequently update. This tool will show you how much regular updates are important. It will point on packages that are being used and can present threats to your system security.
Moreover, some OS don't have a good package manager for all the programs (i.e. developer tools, image and video processing, games, browsers, etc.), and you are required to download the program from the official website the programs. Some programs come with an updater but others don't and they can be a threat to your system security.
arguments -p is system python-pip packages. -o is os packages. -f is file input. -m is for pip3(terminal alias). -t amount of cve to check per name (default is 50), if t < 1 all the entries will be check (Warning: some project have more than thousand CVEs (in particular browsers)). -x fast mode - based only on mitre description, can be far less precise (miss many CVE with bad description for example (apple CVEs and word) and add many with obscure name for example pip packages called rsa|click).
full inspect OS + PIP - python3 app.py
pip: python3 app.py -p
if you have pip and pip3
pip3: python3 app.py -m
OS : python3 app.py -o
File : python3 app.py -f .\example.txt
File format package==version separated with new line (see example.txt)
some programs like web-browser have a lot of CVEs. to limit the amount of cve checked (from the most new one)
Define top slice: python3 app.py -po -t {number}
Fast mode:
python3 app.py -pox -t -1
To review just pip inspection in the following Jupyter notebook (in the attached output, you can see it caught 4 CVEs for my pip in Windows 10 PC)
I ran the script on windows10 (windows10.txt) and ubuntu (ubuntu.txt) - overall it performed better on ubuntu than Windows because of the apt-get package manager which order all my installs with easy and clean naming schema.
Note: when executing on macOS run in terminal export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES before executing the application. This is apparently due to some new security changes made in High Sierra that are breaking lots of Python things that use fork. In our case we use python multiprocessing tool pool.
The implementation is based on getting a program version through existed CLI.
To get python-pip packages' versions I used pip freeze.
The naming and the format are clean and good to use.
After parsing the output, the program search for each pip package in his CVES.
I used https://cve.mitre.org for listing the CVE and https://cve.circl.lu/api/ for detailed JSON on each CVE.
example Mitre API curl https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ycrypto
example Circl API curl https://cve.circl.lu/api/cve/CVE-2018-6594
The JSON contained a field that can help to determine if the CVE is related. I based on it to filter the relevant CVES.
"vulnerable_product": ["cpe:2.3:a:microsoft:skype:8.35:*:*:*:*:android:*:*"]
overall it was fun project and it did found 3 CVEs for pip packages on my Windows 10 PC (view notebook safe_versions.ipynb).
On my ubuntu WSL it found 6 apt-packages' CVEs and 8 pip packages' CVEs (view file ubuntu.txt).
I thought to add a support to get the OS installed programs' name and version. I added support to get Linux (tested on Ubuntu18) installed packages
~$ dpkg -l
Also I added support to get macOS (tested on macOS Catalina) installed packages
The applications from .dmg and the store are saved in this directory
/Applications each application is a directory, the directory name is ending with .app
each application directory has a xml file indicating the version and additional info Applications/{appname}/Contents/Info.plist
In addition, I added support to get windows (tested on Windows10) installed programs To get installed programs
PS ~ Get-ItemProperty HKLM:\\Software\\Wow6432Node\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\*
To get more installed programs
PS ~ Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\*
To get Nicrosoft Store programs
PS ~ Get-AppxPackage
!!! Yet this is not full list of installed program on my Windows 10 !!! for example I did not find Zoom installation with this API.
For each one, I piped the input through fields filter and table format preinstalled programs of PowerShell. for example :
PS ~ Get-AppxPackage | Select-Object DisplayName, DisplayVersion, Publisher, InstallDate | Format-Table -AutoSize
I had problems to fully use the API https://cve.circl.lu/api/. Although they documented support of searching by name it did not work for me. So I search for each program the CVES on https://cve.mitre.org and scrap the HTML data. Then I got detailed JSON information about the CVE from the Circl API.
Some CVEs do not contain all the information about effected versions. But they do write about it in the description.
Different versioning exists: 2.7.15~rc1-1, 2:3.3.12-3ubuntu1.2 (pay attention to :), 0.22.2.post1, 5.26.1-6ubuntu0.3
Programs can have similar names but they are manufactured by the different manufacturers, for example, regular Git and Jenkins Git plugin. It is hard to know the correct name of the product and manufacturer since there is no constant naming and it is not always matched to the info in the CVEs database. In particular, this problem can cause this program to have false positives CVEs.
For example on Ubuntu cpe:2.3:a:git:git:1.8.2.3:::::::* cpe:2.3:a:jenkins:git:1.0.0:::::jenkins::
And on windows 'Git' by 'The Git Development Community'
Even when I got the information from the Circl API, I had to validate the full name and the full version of the Installed program appear in the current CVE. Which is sometimes a problem because some program writes their name with 'version' and architecture such as 'amd64' or 'x64' which added unwanted results in Mitre API.
Especially with Windows installations. Usually, the programs are not installed with a nice package manager. Each installation has a different naming scheme and some have missing versions. Some naming schemas include architecture and version in the name of the program.
Lastly, on Windows, I had to use multiple API to get just a partial list of Windows installed programs. I did try WMIC, but it never finished running and seems as a bottle-neck to this project.